graph LR
A[Simple<br/>Hypotheses & Methods] --> B[Detailed<br/>Analysis Plans] --> C[Registered Reports<br/>Peer Review]
A --> D[Easy]
B --> E[Medium]
C --> F[Difficult]
Enhancing Transparency and Reproducibility in Language Science
2025-07-16
Goal: Make your language research more credible and transparent
Roettger, T. B. (2021). Preregistration in experimental linguistics:
Applications, challenges, and limitations.
Linguistics, 59(5), 1227–1249.
Why this paper?
Key takeaways to focus on:
Preregistration refers to posting a timestamped outline
of the research questions, hypotheses, method, and analysis plan
for a specific project prior to data collection and/or analysis
Key principle: Distinguish between:
graph LR
A[Simple<br/>Hypotheses & Methods] --> B[Detailed<br/>Analysis Plans] --> C[Registered Reports<br/>Peer Review]
A --> D[Easy]
B --> E[Medium]
C --> F[Difficult]
Preregistrations can vary from simple outlines to comprehensive analysis plans with pre-written code
Result: Scientific record biased toward positive findings
Consequence: False positives may mislead theory development
Recent findings from our field:
Evidence: Roettger (2021) documents widespread analytical flexibility in linguistics
Reference: Roettger, T. B. (2021). Preregistration in experimental linguistics:
Applications, challenges, and limitations. Linguistics, 59(5), 1227-1249.
Key insight: Preregistration is flexible and adaptable to linguistic research
Goal: Be specific enough that a skeptical reader is convinced you planned ahead
🔗 Resource: OSF Templates
Linguistics-specific: Secondary data template
Focus on the essentials:
Good for: Beginners, exploratory studies, time constraints
Include specifics:
Good for: Confirmatory studies, complex designs
Two-stage process:
OSF (Open Science Framework) - Most comprehensive - Multiple templates - Integration with project management - Embargos up to 4 years
AsPredicted - Simple and quick - 8 basic questions - Good for beginners - Free to use
| Feature | OSF | AsPredicted |
|---|---|---|
| Templates | Many | One (9 questions) |
| Output | Web page | PDF with URL |
| Embargo | 4 years | Private option |
| Collaboration | Multi-author | Email approval |
| Cost | Free | Free |
| Integration | Project management | Standalone |
gantt
title Preregistration Timeline
dateFormat YYYY-MM-DD
section Planning
Draft preregistration :a1, 2025-01-01, 2w
Advisor review :a2, after a1, 1w
Revisions :a3, after a2, 1w
section Execution
Register study :b1, after a3, 1d
Data collection :b2, after b1, 8w
Analysis :b3, after b2, 4w
When things don’t go as planned:
Remember: Transparency is the goal, not perfect adherence
Standard language for reporting deviations:
Key principle: Transparency, not perfection
Scenario: You preregistered a study but encountered problems:
Discussion: How would you handle each?
The 9 AsPredicted Questions:
Result: Time-stamped PDF with unique URL for verification
Research Question: How does prosodic prominence
affect syntactic processing in German?
Hypothesis: Prominent words will show faster integration into syntactic structure
Participants: 40 German native speakers, 18-35 years, no language disorders
Materials: 120 sentences with prominence manipulation, normed for frequency/length
Procedure: Self-paced reading + comprehension questions
Analysis: Linear mixed-effects models with prominence as fixed factor
Exclusions: Accuracy <80% on comprehension, reading times >3 SDs
Research Question: Does word predictability
affect pronunciation in spontaneous speech?
Data: HCRC Map Task Corpus (Anderson et al., 1991)
Preregistered decisions: - Predictability measure: Trigram probability from Google Books - Acoustic measure: Mean F0 of vowel nucleus - Control variables: Speaker sex, utterance position, word frequency - Exclusions: Function words, words <3 phonemes - Model: Linear mixed-effects: F0 ~ predictability + controls + (1|speaker)
Key insight: Even with existing data, many analytical choices remain
Roettger’s key insight: “Preregistration is not a panacea for all problems,
but it’s a practice we can integrate into our work flow right away”
Immediate action: Choose one upcoming study to preregister
Essential websites: - Center for Open Science: cos.io/prereg - OSF Preregistration: osf.io/prereg - AsPredicted: aspredicted.org - Templates: osf.io/zab38 - Registered Reports: cos.io/rr
Reading recommendations: - The Preregistration Revolution (Nosek et al., 2018) - Research Preregistration 101 (APS) - A manifesto for reproducible science (Munafò et al., 2017)
Small group task (10 minutes):
Scenarios provided: - Prosodic prominence and sentence processing - Cross-linguistic comparison of word order effects - Bilingual language switching patterns - Corpus analysis of discourse markers
What challenges do you see for preregistering linguistic research?
How might preregistration help your current SFB 1252 project?
Who in your research area could be your accountability partner?
Contact: job.schepens@uni-koeln.de | Project S, SFB 1252 Workshop Materials: Available on SFB 1252 OSF project
RDM Workshop - SFB 1252 Prominence in Language